Extraordinary amounts of data are being produced in many branches of science.Proven statistical methods are no longer applicable with extraordinary largedata sets due to computational limitations. A critical step in big dataanalysis is data reduction. Existing investigations in the context of linearregression focus on subsampling-based methods. However, not only is thisapproach prone to sampling errors, it also leads to a covariance matrix of theestimators that is typically bounded from below by a term that is of the orderof the inverse of the subdata size. We propose a novel approach, termedinformation-based optimal subdata selection (IBOSS). Compared to leadingexisting subdata methods, the IBOSS approach has the following advantages: (i)it is significantly faster; (ii) it is suitable for distributed parallelcomputing; (iii) the variances of the slope parameter estimators converge to 0as the full data size increases even if the subdata size is fixed, i.e., theconvergence rate depends on the full data size; (iv) data analysis for IBOSSsubdata is straightforward and the sampling distribution of an IBOSS estimatoris easy to assess. Theoretical results and extensive simulations demonstratethat the IBOSS approach is superior to subsampling-based methods, sometimes byorders of magnitude. The advantages of the new approach are also illustratedthrough analysis of real data.
展开▼
